CITY OF FAIRFIELD 


RESOLUTION NO. 2017 - 279 


RESOLUTION OF THE CITY COUNCIL OF THE CITY OF FAIRFIELD APPROVING 
AMENDMENT NO. 1 TO PROFESSIONAL SERVICES AGREEMENT BETWEEN THE 
CITY OF FAIRFIELD AND KENNEDY/JENKS CONSULTANTS FOR ENGINEERING 
SERVICES DURING CONSTRUCTION FOR THE SCADA SYSTEM IMPROVEMENTS 
PROJECT AT THE NORTH BAY REGIONAL WATER TREATMENT PLANT 


WHEREAS, the North Bay Regional Water Treatment Plant is a water treatment facility 
jointly owned by the cities of Fairfield and Vacaville; and 

WHEREAS, staff would like to amend the City’s Professional Services Agreement with 
Kennedy/Jenks Consultants to provide additional engineering support services for post 
project testing of the Supervisory Control and Data Acquisition (SCADA) system; and 

WHEREAS, this Amendment is only to change the scope of the work to be performed 
and will not result in any increase in the original contract amount; and 

WHEREAS, staff is requesting these additional professional engineering technical 
services from Kennedy/Jenks Consultants for this assistance. 

NOW, THEREFORE, THE COUNCIL OF THE CITY OF FAIRFIELD HEREBY 
RESOLVES: 

1. The City Manager is hereby authorized and directed to execute on behalf of the City 
of Fairfield Amendment No. 1 to the Agreement between the City of Fairfield and 
Kennedy/Jenks Consultants for Engineering Services During Construction for the 
SCADA System Improvements Project at the North Bay Regional Water Treatment 
Plant. 







PASSED AND ADOPTED this day of 


AYES: COUNCILMEMBERS: _Pj 

NOES; COUNCILMEMBERS: _ 

ABSENT: COUNCILMEMBERS: _ 

ABSTAIN: COUNCILMEMBERS: _ 

MAYO 
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AMENDMENT NO. 1 


TO A CONSULTANT SERVICES AGREEMENT BETWEEN THE CITY 
OF FAIRFIELD AND KENNEDY/JENKS CONSULTANTS FOR THE 
NORTH BAY REGIONAL WATER TREATMENT PLANT SCADA 
IMPROVEMENTS PROJECT - ENGINEERING SERVICES DURING 

CONSTRUCTION 


This AMENDMENT No. 1 (hereinafter “AMENDMENT”) to the Agreement between 
the City of Fairfield and Kennedy/Jenks Consultants for the North Bay Regional Water 
Treatment Plant SCADA Improvements Project - Engineering Service During 

Construction, dated August 23, 2016, is made and entered this I / day of_ 

2 - 0 1'? , 26 TT 7 by and between the CITY OF FAIRFIELD, a municipal 
corporation (hereinafter “CITY”), and KENNEDY/JENKS CONSULTANTS (hereinafter 
“CONSULTANT”). 


WITNESSETH 


WHEREAS, on August 23, 2016, the parties entered into an AGREEMENT 
for Consulting Services for the Engineering Services During Construction of the North 
Bay Regional Water Treatment Plant SCADA Improvements Project (PROJECT): and 

WHEREAS, CONSULTANT has agreed to modifications to the 
AGREEMENT to include additional work scope items with no fee increase. 

NOW, THEREFORE, in consideration of the mutual promises, covenants 
and agreements herein set forth, the parties do hereby agree as follows: 


1. Section 1) SERVICES of the AGREEMENT shall be amended to include the 
following: 

Add to Exhibit A - Scope of Services the following Task 5: 


“Task 5 - Validation and Testing 

This Task is intended to provide validation of the following areas of technical infrastructure 
implemented at the NBR Water Treatment Plant in accordance with Specification Section 17340 
paragraph 3.08 and to provide CSET reporting and review: 

1. Virtual Machine High Availability Validation 

2. Disaster Recovery Validation 



3. Cyber-Security Testing and Validation 

4. CSET Reporting and Review 

For the first 2 cases, the validation will include developing a test plan, and monitoring the 
execution of the test plan by NBR IT Staff. For the Cyber-Security testing, the execution of the 
plan will be handled by CONSULTANT’S sub-consultant, AllConnected, with the option to be on 
site for the internal penetration testing. 

5.1 - Virtualization Resiliency Validation and Testing: 

a) Hold Task Kickoff Meeting with NBR and City IT Staff at NBRWTP including meeting 
preparation. While there, audit SCADA workstations and servers for Antivirus, 
Windows firewalling, and event logging and perform internal vulnerability testing with 
non-disruptive penetration testing by scanning all internal networks related to 
NBRWTP with Nessus tool. 

b) Review VMWare and Storage Area Network (SAN) configuration for path and 
component failure. Determine from IT staff what the expected results are during 
specific component failure. 

c) Assess Virtual Machine (VM) Usage of physical resources, such as CPU, Memory, 
and Disk to determine High Availability (HA) capacity of the cluster. 

d) Participate in teleconference with NBR and City IT Staff to determine changes that 
will be needed prior to failover test. 

e) Create test script that indicates which components will be forced into failure and in 
what order. 

f) Incorporate test plan review comments. 

g) Oversee the resiliency test based on the test script. 

h) Prepare summary memorandum regarding testing and validation performed. 

5.2 - Disaster Recovery Validation and Testing 

a) Review Zerto configuration and VM Selection. Determine from IT Staff what the 
expected VM and Network subnet failover should be. Document the IT Staff 
expectations. 

b) Assess Bandwidth and Hardware resource availability from the NBR to City Hall. 

c) Create Disaster Recovery (DR) Testing document that details how a server resource 
failure would be handled, focusing on the manual steps required for a successful 
failover test. 

d) Incorporate test plan review comments. 

e) Work with NBR IT to perform failover test to City Hall simulating a complete failure of 
the storage and server infrastructure at the WTP. 

f) Document findings during failover and document tailback procedures. 

g) If warranted, work with NBR IT Staff to perform a second failover test based on 
findings during the first test. 

5.3 - Cyber-Security Validation and Testing 

a) Review as-built documentation for conformance with Drawing I006-Rev3. 



b) Collect inventory of external and internal IP Subnets that will be tested. 

c) Audit firmware versions and configuration details of Layer 3 Gateway, Sonicwall 
Firewall, NBR IT Switch (HP 5406), and Stratix Switch #1 (5410). 

d) Perform external vulnerability testing with non-disruptive penetration testing. 

e) Provide raw data from the vulnerability tests in Excel format to the NBR IT Staff for 
review and mitigation. 

f) Document members of the security response team and create cyber response 
workflow. 

g) Participate in summary teleconference to review findings and next steps. 

5.4 - Documentation 

a) Create document for any recommended changes to: 1) the virtual environment to 
increase resiliency; 2) the Zerto or Network configuration to ease failover to the City Hall 
infrastructure; and 3) remove vulnerabilities and follow security best practices. 

b) Prepare CSET questionnaire with "Key Questions", "NIST 800-53 v3", "NIST 800-82", 
and "NIST 800-82 vl” and all pertinent City details, create simple diagram of the 
NBRWTP Environment for the CSET tool. 

c) Work with NBR IT to collect answers to the CSET questionnaire from the appropriate 
parties to a reasonable extent. 

d) Fill in gaps in the CSET questionnaire based upon the analysis performed during Task 
5.3. 

e) Perform CSET Internal review to validate responses and submit exception report to NBR 
IT for remediation. 

f) Participate in CSET NBRWTP Review teleconference with City IT Department and key 
NBRWTP personnel to review final document. 

g) Prepare summary memorandum regarding CSET review of SCADA System. 

Assistance from City 

• Sufficient access will be provided to CONSULTANT by CITY to the physical and 
logical infrastructure to perform the tasks described in Task 5. 

• VPN Access will be available for Nick Peros and Richard Pressler to enable access 
to the critical systems that will be tested and verified. 

• Staff from City IT, NBRWTP Operations, and CITY’S Systems Integration Contractor 
will be available for support during execution of the Task 5 work. 

• City IT Staff will contribute approximately 2 days to filling out the CSET 
questionnaire. CSET questionnaire completion will be limited to SCADA systems and 
elements related to the SCADA systems (example: internet access as it pertains to 
inbound and outbound connectivity with regards to SCADA systems only will be 
addressed, not to the administrative components at City Hall or the WTP) 

• A jump server or laptop (may be virtual) will be made available to CONSULTANT by 
CITY throughout the Task 5 work in order to run tools inside of the WTP 
environment.” 



2. Existing Task budgets shall be re-allocated according to the table attached hereto 
as Exhibit A-1 in order to accomplish the additional work. 

3. All work, including Task 5 work, shall be fully completed before June 30, 2018. 

4. Except as specifically set forth herein, all terms and conditions of the 
AGREEMENT between the CITY and CONSULTANT, dated August 23, 2016, 
shall remain in full force and effect. 

IN WITNESS WHEREOF, the parties hereto have executed this Amendment on 
_Li_, 2et7. 

EXECUTED as of the day first above-stated. 

City of Fairfield, a municipal corporation Kennedy/Jenks Consultants 

Douglas B. Henderson 
Vice President_ 



APPROVED AS TO FORM: 




CiTy 


By: 


Its: 



EXHIBIT A-1 


City of Fairfield 

NBRWTP - SCADA System Improvements - ESDC 
Amendment No. 1 - Budget Re-allocations 


Task No. Task Description 

Original 

Revised 

Contract 

Contract 

Amount 

Amount 

1 Construction Management Assistance 

1.1 Communications and Information Management 

1.2 Attend and Document Final Inspection 

2 Construction Observation 

2. 1 Servers and HMI Installations and Testing 

3 Engineering Services During Construction 

3. 1 Meetings Participation 

3.2 Submittal Reviews 

3.3 Requests for Information (RFI) Responses 

3.4 Request for Quotation (RFQ) Preparation 

3.5 Site Visits/Factory Tests 

3.6 Record Drawings Preparation 

4 Project Management 

4.1 Project Set-up and Management 

4.2 Project Status Reporting 

4.3 Quality Assurance/Quality Control (QA/QC) Reviews 

5 Validation and Testing 

5. 1 Virtualization Resiliency Validation and Testing 

5.2 Disaster Recovery Validation and Testing 

5.3 Cyber Security Validation and Testing 

5.4 Documentation 

$12,250.00 

$5,000.00 

$8,237.00 

$4,500.00 

$4,013.00 

$500.00 



$15,950.00 

$500.00 

$15,950.00 

$500.00 



$186,100.00 

$132,810.00 

$23,630.00 

$29,600.00 

$66,393.00 

$29,500.00 

$52,582.00 

$29,500.00 

$11,43800 

$4,200.00 

$21,054.00 

$29,010.00 

$11,003.00 

$11,000.00 



$31,100.00 

$27,640.00 

$9,580.00 

$12,940.00 

$8,208.00 

$6,100.00 

$13,312.00 

$8,600.00 



$0.00 

$79,450.00 

$0,00 

$20,600.00 

$0.00 

$11,000.00 

$0.00 

$16,500.00 

$0.00 

$31,350.00 



Totals 

$245,400.00 

$245,400.00 
























































